Massive simultaneous remote digital presence world

ABSTRACT

Various methods and apparatus are described herein for enabling one or more users to interface with virtual or augmented reality environments. An example system includes a computing network having computer servers interconnected through high bandwidth interfaces to gateways for processing data and/or for enabling communication of data between the servers and one or more local user interface devices. The servers include memory, processing circuitry, and software for designing and/or controlling virtual worlds, as well as for storing and processing user data and data provided by other components of the system. One or more virtual worlds may be presented to a user through a user device for the user to experience and interact. A large number of users may each use a device to simultaneously interface with one or more digital worlds by using the device to observe and interact with each other and with objects produced within the digital worlds.

CROSS-REFERENCE TO RELATED APPLICATIONS

The application is a continuation of U.S. patent application Ser. No.13/465,682, filed on May 7, 2012, which pursuant to 35 U.S.C. § 119(e),claims priority from and the benefit of, and hereby incorporates byreference for all purposes, U.S. Provisional Patent Application Ser. No.61/483,505, filed May 6, 2011, and U.S. Provisional Patent ApplicationSer. No. 61/483,511, filed May 6, 2011. This application also claimspriority pursuant to 35 U.S.C. § 119(a) to PCT Application Serial No.PCT/US2012/036681, filed May 4, 2012.

FIELD OF THE INVENTION

This invention generally relates to methods and apparatus for enablinginteractive virtual or augmented reality environments for multipleusers.

BACKGROUND

Virtual and augmented reality environments are generated by computersusing, in part, data that describes the environment. This data maydescribe, for example, various objects with which a user may sense andinteract with. Examples of these objects include objects that arerendered and displayed for a user to see, audio that is played for auser to hear, and tactile (or haptic) feedback for a user to feel. Usersmay sense and interact with the virtual and augmented realityenvironments through a variety of visual, auditory and tactical means.

SUMMARY

The present disclosure describes various systems and methods forenabling one or more users to interface with or participate in virtualor augmented reality environments.

In one exemplary embodiment, a system includes a computing networkhaving computer servers interconnected through high bandwidth interfacesto gateways for processing data and/or for enabling communication ofdata between the servers and one or more local user interface devices.The servers include memory, processing circuitry, and software fordesigning and/or controlling virtual worlds, as well as for storing andprocessing user data and data provided by other components of thesystem. One or more virtual worlds may be presented to a user through auser device for the user to experience and interact. A large number ofusers may each use a device to simultaneously interface with one or moredigital worlds by using the device to observe and interact with eachother and with objects produced within the digital worlds.

Examples of user devices include a smart phone, tablet device, heads-updisplay (HUD), gaming console, or generally any other device capable ofcommunicating data and generating or communicating an interface to theuser to see, hear and/or touch. Generally, the user device will includea processor for executing program code stored in memory on the device,coupled with a visual display, and a communications interface. Theinterface enables a visual, audible, and/or physical interaction betweenthe user and a digital world, including other users and objects (real orvirtual) presented to the user. In one embodiment, the user devicecomprises a head-mounted display system having an interface,user-sensing system, environment-sensing system, and a processor.

The foregoing and other features and advantages of the presentdisclosure will become further apparent from the following detaileddescription of exemplary embodiments, read in conjunction with theaccompanying drawings. The detailed description and drawings are merelyillustrative of the disclosure, rather than limiting the scope of theinvention as defined by the appended claims and equivalents thereof.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments are illustrated by way of example in the accompanyingfigures not necessarily drawn to scale, in which like numbers indicatesimilar parts, and in which:

FIG. 1 illustrates a representative embodiment of the disclosed systemfor enabling interactive virtual or augmented reality environments formultiple users;

FIG. 2 illustrates an example of a user device for interacting with thesystem illustrated in FIG. 1;

FIG. 3 illustrates an example embodiment of a mobile, wearable userdevice;

FIG. 4 illustrates an example of objects viewed by a user when themobile, wearable user device of FIG. 3 is operating in an augmentedmode;

FIG. 5 illustrates an example of objects viewed by a user when themobile, wearable user device of FIG. 3 is operating in a virtual mode;

FIG. 6 illustrates an example of objects viewed by a user when themobile, wearable user device of FIG. 3 is operating in a blended virtualinterface mode;

FIG. 7 illustrates an embodiment wherein two users located m differentgeographical locations each interact with the other user and a commonvirtual world through their respective user devices;

FIG. 8 illustrates an embodiment wherein the embodiment of FIG. 7 isexpanded to include the use of a haptic device;

FIG. 9A illustrates an example of mixed mode interfacing, wherein afirst user is interfacing a digital world in a blended virtual interfacemode and a second user is interfacing the same digital world in avirtual reality mode;

FIG. 9B illustrates another example of mixed mode interfacing, whereinthe first user is interfacing a digital world in a blended virtualinterface mode and the second user is interfacing the same digital worldin an augmented reality mode;

FIG. 10 illustrates an example illustration of a user's view wheninterfacing the system in an augmented reality mode; and

FIG. 11 illustrates an example illustration of a user's view showing avirtual object triggered by a physical object when the user isinterfacing the system in an augmented reality mode.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Referring to FIG. 1, system 100 is representative hardware forimplementing processes described below. This representative systemcomprises a computing network 105 comprised of one or more computerservers 110 connected through one or more high bandwidth interfaces 115.The servers in the computing network need not be co-located. The one ormore servers 110 each comprise one or more processors for executingprogram instructions. The servers also include memory for storing theprogram instructions and data that is used and/or generated by processesbeing carried out by the servers under direction of the programinstructions.

The computing network 105 communicates data between the servers 110 andbetween the servers and one or more user devices 120 over one or moredata network connections 130. Examples of such data networks include,without limitation, any and all types of public and private datanetworks, both mobile and wired, including for example theinterconnection of many of such networks commonly referred to as theInternet. No particular media, topology or protocol is intended to beimplied by the figure.

User devices are configured for communicating directly with computingnetwork 105, or any of the servers 110. Alternatively, user devices 120communicate with the remote servers 110, and, optionally, with otheruser devices locally, through a specially programmed, local gateway 140for processing data and/or for communicating data between the network105 and one or more local user devices 120.

As illustrated, gateway 140 is implemented as a separate hardwarecomponent, which includes a processor for executing softwareinstructions and memory for storing software instructions and data. Thegateway has its own wired and/or wireless connection to data networksfor communicating with the servers 110 comprising computing network 105.Alternatively, gateway 140 can be integrated with a user device 120,which is worn or carried by a user. For example, the gateway 140 may beimplemented as a downloadable software application installed and runningon a processor included in the user device 120. The gateway 140provides, in one embodiment, one or more users access to the computingnetwork 105 via the data network 130.

Servers 110 each include, for example, working memory and storage forstoring data and software programs, microprocessors for executingprogram instructions, graphics processors and other special processorsfor rendering and generating graphics, images, video, audio andmulti-media files. Computing network 105 may also comprise devices forstoring data that is accessed, used or created by the servers 110.

Software programs running on the servers and optionally user devices 120and gateways 140, are used to generate digital worlds (also referred toherein as virtual worlds) with which users interact with user devices120. A digital world is represented by data and processes that describeand/or define virtual, non-existent entities, environments, andconditions that can be presented to a user through a user device 120 forusers to experience and interact with. For example, some type of object,entity or item that will appear to be physically present wheninstantiated in a scene being viewed or experienced by a user mayinclude a description of its appearance, its behavior, how a user ispermitted to interact with it, and other characteristics. Data used tocreate an environment of a virtual world (including virtual objects) mayinclude, for example, atmospheric data, terrain data, weather data,temperature data, location data, and other data used to define and/ordescribe a virtual environment. Additionally, data defining variousconditions that govern the operation of a virtual world may include, forexample, laws of physics, time, spatial relationships and other datathat may be used to define and/or create various conditions that governthe operation of a virtual world (including virtual objects).

The entity, object, condition, characteristic, behavior or other featureof a digital world will be generically referred to herein, unless thecontext indicates otherwise, as an object (e.g., digital object, virtualobject, rendered physical object, etc.). Objects may be any type ofanimate or inanimate object, including but not limited to, buildings,plants, vehicles, people, animals, creatures, machines, data, video,text, pictures, and other users. Objects may also be defined in adigital world for storing information about items, behaviors, orconditions actually present in the physical world. The data thatdescribes or defines the entity, object or item, or that stores itscurrent state, is generally referred to herein as object data. This datais processed by the servers 110 or, depending on the implementation, bya gateway 140 or user device 120, to instantiate an instance of theobject and render the object in an appropriate manner for the user toexperience through a user device.

Programmers who develop and/or curate a digital world create or defineobjects, and the conditions under which they are instantiated. However,a digital world can allow for others to create or modify objects. Oncean object is instantiated, the state of the object may be permitted tobe altered, controlled or manipulated by one or more users experiencinga digital world.

For example, in one embodiment, development, production, andadministration of a digital world is generally provided by one or moresystem administrative programmers. In some embodiments, this may includedevelopment, desi_(gn), and/or execution of story lines, themes, andevents in the digital worlds as well as distribution of narrativesthrough various forms of events and media such as, for example, film,digital, network, mobile, augmented reality, and live entertainment. Thesystem administrative programmers may also handle technicaladministration, moderation, and curation of the digital worlds and usercommunities associated therewith, as well as other tasks typicallyperformed by network administrative personnel.

Users interact with one or more digital worlds using some type of alocal computing device, which is generally designated as a user device120. Examples of such user devices include, but are not limited to, asmart phone, tablet device, heads-up display (HUD), gaming console, orany other device capable of communicating data and providing aninterface or display to the user, as well as combinations of suchdevices. In some embodiments, the user device 120 may include, orcommunicate with, local peripheral or input/output components such as,for example, a keyboard, mouse, joystick, gaming controller, hapticinterface device, motion capture controller, audio equipment, voiceequipment, projector system, 3D display, and holographic 3D contactlens.

An example of a user device 120 for interacting with the system 100 isillustrated in FIG. 2. In the example embodiment shown in FIG. 2, a user210 may interface one or more digital worlds through a smart phone 220.The gateway is implemented by a software application 230 stored on andrunning on the smart phone 220. In this particular example, the datanetwork 130 includes a wireless mobile network connecting the userdevice (i.e., smart phone 220) to the computer network 105.

In one implementation of preferred embodiment, system 100 is capable ofsupporting a large number of simultaneous users (e.g., millions ofusers), each interfacing with the same digital world, or with multipledigital worlds, using some type of user device 120.

The user device provides to the user an interface for enabling a visual,audible, and/or physical interaction between the user and a digitalworld generated by the servers 110, including other users and objects(real or virtual) presented to the user. The interface provides the userwith a rendered scene that can be viewed, heard or otherwise sensed, andthe ability to interact with the scene in real-time. The manner in whichthe user interacts with the rendered scene may be dictated by thecapabilities of the user device. For example, if the user device is asmart phone, the user interaction may be implemented by a usercontacting a touch screen. In another example, if the user device is acomputer or gaming console, the user interaction may be implementedusing a keyboard or gaming controller. User devices may includeadditional components that enable user interaction such as sensors,wherein the objects and information (including gestures) detected by thesensors may be provided as input representing user interaction with thevirtual world using the user device.

The rendered scene can be presented in various formats such as, forexample, two-dimensional or three-dimensional visual displays (includingprojections), sound, and haptic or tactile feedback. The rendered scenemay be interfaced by the user in one or more modes including, forexample, augmented reality, virtual reality, and combinations thereof.The format of the rendered scene, as well as the interface modes, may bedictated by one or more of the following: user device, data processingcapability, user device connectivity, network capacity and systemworkload. Having a large number of users simultaneously interacting withthe digital worlds, and the real-time nature of the data exchange, isenabled by the computing network 105, servers 110, the gateway component140 (optionally), and the user device 120.

In one example, the computing network 105 is comprised of a large-scalecomputing system having single and/or multi-core servers (i.e., servers110) connected through high-speed connections (e.g., high bandwidthinterfaces 115). The computing network 105 may form a cloud or gridnetwork. Each of the servers includes memory, or is coupled withcomputer-readable memory for storing software for implementing data tocreate, design, alter, or process objects of a digital world. Theseobjects and their instantiations may be dynamic, come in and out ofexistence, change over time, and change in response to other conditions.Examples of dynamic capabilities of the objects are generally discussedherein with respect to various embodiments. In some embodiments, eachuser interfacing the system 100 may also be represented as an object,and/or a collection of objects, within one or more digital worlds.

The servers 110 within the computing network 105 also storecomputational state data for each of the digital worlds. Thecomputational state data (also referred to herein as state data) may bea component of the object data, and generally defines the state of aninstance of an object at a given instance in time. Thus, thecomputational state data may change over time and may be impacted by theactions of one or more users and/or programmers maintaining the system100. As a user impacts the computational state data (or other datacomprising the digital worlds), the user directly alters or otherwisemanipulates the digital world. If the digital world is shared with, orinterfaced by, other users, the actions of the user may affect what isexperienced by other users interacting with the digital world. Thus, insome embodiments, changes to the digital world made by a user will beexperienced by other users interfacing with the system 100.

The data stored in one or more servers 110 within the computing network105 is, in one embodiment, transmitted or deployed at a high-speed, andwith low latency, to one or more user devices 120 and/or gatewaycomponents 140. In one embodiment, object data shared by servers may becomplete or may be compressed, and contain instructions for recreatingthe full object data on the user side, rendered and visualized by theuser's local computing device (e.g., gateway 140 and/or user device120). Software running on the servers 110 of the computing network 105may, in some embodiments, adapt the data it generates and sends to aparticular user's device 120 for objects within the digital world (orany other data exchanged by the computing network 105) as a function ofthe user's specific device and bandwidth. For example, when a userinteracts with a digital world through a user device 120, a server 110may recognize the specific type of device being used by the user, thedevice's connectivity and/or available bandwidth between the user deviceand server, and appropriately size and balance the data being deliveredto the device to optimize the user interaction. An example of this mayinclude reducing the size of the transmitted data to a low resolutionquality, so that the data may be displayed on a particular user devicehaving a low resolution display. In a preferred embodiment, thecomputing network 105 and/or gateway component 140 deliver data to theuser device 120 at a rate sufficient to present an interface operatingat 15 frames/second or higher, and at a resolution that is highdefinition quality or greater.

The gateway 140 provides local connection to the computing network 105for one or more users. In some embodiments, it may be implemented by adownloadable software application that runs on the user device 120 oranother local device, such as that shown in FIG. 2. In otherembodiments, it may be implemented by a hardware component (withappropriate software/firmware stored on the component, the componenthaving a processor) that is either in communication with, but notincorporated with or attracted to, the user device 120, or incorporatedwith the user device 120. The gateway 140 communicates with thecomputing network 105 via the data network 130, and provides dataexchange between the computing network 105 and one or more local userdevices 120. As discussed in greater detail below, the gateway component140 may include software, firmware, memory, and processing circuitry,and may be capable of processing data communicated between the network105 and one or more local user devices 120.

In some embodiments, the gateway component 140 monitors and regulatesthe rate of the data exchanged between the user device 120 and thecomputer network 105 to allow optimum data processing capabilities forthe particular user device 120. For example, in some embodiments, thegateway 140 buffers and downloads both static and dynamic aspects of adigital world, even those that are beyond the field of view presented tothe user through an interface connected with the user device. In such anembodiment, instances of static objects (structured data, softwareimplemented methods, or both) may be stored in memory (local to thegateway component 140, the user device 120, or both) and are referencedagainst the local user's current position, as indicated by data providedby the computing network 105 and/or the user's device 120. Instances ofdynamic objects, which may include, for example, intelligent softwareagents and objects controlled by other users and/or the local user, arestored in a high-speed memory buffer. Dynamic objects representing atwo-dimensional or three-dimensional object within the scene presentedto a user can be, for example, broken down into component shapes, suchas a static shape that is moving but is not changing, and a dynamicshape that is changing. The part of the dynamic object that is changingcan be updated by a real-time, threaded high priority data stream from aserver 110, through computing network 105, managed by the gatewaycomponent 140. As one example of a prioritized threaded data stream,data that is within a 60 degree field-of-view of the user's eye may begiven higher priority than data that is more peripheral. Another exampleincludes prioritizing dynamic characters and/or objects within theuser's field-of-view over static objects in the background.

In addition to managing a data connection between the computing network105 and a user device 120, the gateway component 140 may store and/orprocess data that may be presented to the user device 120. For example,the gateway component 140 may, in some embodiments, receive compresseddata describing, for example, graphical objects to be rendered forviewing by a user, from the computing network 105 and perform advancedrendering techniques to alleviate the data load transmitted to the userdevice 120 from the computing network 105. In another example, in whichgateway 140 is a separate device, the gateway 140 may store and/orprocess data for a local instance of an object rather than transmittingthe data to the computing network 105 for processing.

Referring now also to FIG. 3, the digital worlds may be experienced byone or more users in various formats that may depend upon thecapabilities of the user's device. In some embodiments, the user device120 may include, for example, a smart phone, tablet device, heads-updisplay (HUD), gaming console, or a wearable device. Generally, the userdevice will include a processor for executing program code stored inmemory on the device, coupled with a display, and a communicationsinterface. An example embodiment of a user device is illustrated in FIG.3, wherein the user device comprises a mobile, wearable device, namely ahead-mounted display system 300. In accordance with an embodiment of thepresent disclosure, the head-mounted display system 300 includes a userinterface 302, user-sensing system 304, environment-sensing system 306,and a processor 308. Although the processor 308 is shown in FIG. 3 as anisolated component separate from the head-mounted system 300, in analternate embodiment, the processor 308 may be integrated with one ormore components of the head-mounted system 300, or may be integratedinto other system 100 components such as, for example, the gateway 140.

The user device presents to the user an interface 302 for interactingwith and experiencing a digital world. Such interaction may involve theuser and the digital world, one or more other users interfacing thesystem 100, and objects within the digital world. The interface 302generally provides image and/or audio sensory input (and in someembodiments, physical sensory input) to the user. Thus, the interface302 may include speakers (not shown) and a display component 303capable, in some embodiments, of enabling stereoscopic 3D viewing and/or3D viewing which embodies more natural characteristics of the humanvision system. In some embodiments, the display component 303 maycomprise a transparent interface (such as a clear OLED) which, when inan “off” setting, enables an optically correct view of the physicalenvironment around the user with little-to-no optical distortion orcomputing overlay. As discussed in greater detail below, the interface302 may include additional settings that allow for a variety ofvisual/interface performance and functionality.

The user-sensing system 304 may include, in some embodiments, one ormore sensors 310 operable to detect certain features, characteristics,or information related to the individual user wearing the system 300.For example, in some embodiments, the sensors 310 may include a cameraor optical detection/scanning circuitry capable of detecting real-timeoptical characteristics/measurements of the user such as, for example,one or more of the following: pupil constriction/dilation, angularmeasurement/positioning of each pupil, sphericity, eye shape (as eyeshape changes over time) and other anatomic data. This data may provide,or be used to calculate, information (e.g., the user's visual focalpoint) that may be used by the head-mounted system 300 and/or interfacesystem 100 to optimize the user's viewing experience. For example, inone embodiment, the sensors 310 may each measure a rate of pupilcontraction for each of the user's eyes. This data may be transmitted tothe processor 308 (or the gateway component 140 or to a server 110),wherein the data is used to determine, for example, the user's reactionto a brightness setting of the interface display 303. The interface 302may be adjusted in accordance with the user's reaction by, for example,dimming the display 303 if the user's reaction indicates that thebrightness level of the display 303 is too high. The user-sensing system304 may include other components other than those discussed above orillustrated in FIG. 3. For example, in some embodiments, theuser-sensing system 304 may include a microphone for receiving voiceinput from the user. The user sensing system may also include one ormore infrared camera sensors, one or more visible spectrum camerasensors, structured light emitters and/or sensors, infrared lightemitters, coherent light emitters and/or sensors, gyros, accelerometers,magnetometers, proximity sensors, GPS sensors, ultrasonic emitters anddetectors and haptic interfaces.

The environment-sensing system 306 includes one or more sensors 312 forobtaining data from the physical environment around a user. Objects orinformation detected by the sensors may be provided as input to the userdevice. In some embodiments, this input may represent user interactionwith the virtual world. For example, a user viewing a virtual keyboardon a desk may gesture with his fingers as if he were typing on thevirtual keyboard. The motion of the fingers moving may be captured bythe sensors 312 and provided to the user device or system as input,wherein the input may be used to change the virtual world or create newvirtual objects. For example, the motion of the fingers may berecognized (using a software program) as typing, and the recognizedgesture of typing may be combined with the known location of the virtualkeys on the virtual keyboard. The system may then render a virtualmonitor displayed to the user (or other users interfacing the system)wherein the virtual monitor displays the text being typed by the user.

The sensors 312 may include, for example, a generally outward-facingcamera or a scanner for interpreting scene information, for example,through continuously and/or intermittently projected infrared structuredlight. The environment-sensing system 306 may be used for mapping one ormore elements of the physical environment around the user by detectingand registering the local environment, including static objects, dynamicobjects, people, gestures and various lighting, atmospheric and acousticconditions. Thus, in some embodiments, the environment-sensing system306 may include image-based 3D reconstruction software embedded in alocal computing system (e.g., gateway component 140 or processor 308)and operable to digitally reconstruct one or more objects or informationdetected by the sensors 312. In one exemplary embodiment, theenvironment-sensing system 306 provides one or more of the following:motion capture data (including gesture recognition), depth sensing,facial recognition, object recognition, unique object featurerecognition, voice/audio recognition and processing, acoustic sourcelocalization, noise reduction, infrared or similar laser projection, aswell as monochrome and/or color CMOS sensors (or other similar sensors),field-of-view sensors, and a variety of other optical-enhancing sensors.It should be appreciated that the environment-sensing system 306 mayinclude other components other than those discussed above or illustratedin FIG. 3. For example, in some embodiments, the environment-sensingsystem 306 may include a microphone for receiving audio from the localenvironment. The user sensing system may also include one or moreinfrared camera sensors, one or more visible spectrum camera sensors,structure light emitters and/or sensors, infrared light emitters,coherent light emitters and/or sensors gyros, accelerometers,magnetometers, proximity sensors, GPS sensors, ultrasonic emitters anddetectors and haptic interfaces.

As mentioned above, the processor 308 may, in some embodiments, beintegrated with other components of the head-mounted system 300,integrated with other components of the interface system 100, or may bean isolated device (wearable or separate from the user) as shown in FIG.3. The processor 308 may be connected to various components of thehead-mounted system 300 and/or components of the interface system 100through a physical, wired connection, or through a wireless connectionsuch as, for example, mobile network connections (including cellulartelephone and data networks), Wi-Fi or Bluetooth. The processor 308 mayinclude a memory module, integrated and/or additional graphicsprocessing unit, wireless and/or wired internet connectivity, and codecand/or firmware capable of transforming data from a source (e.g., thecomputing network 105, the user-sensing system 304, theenvironment-sensing system 306, or the gateway component 140) into imageand audio data, wherein the images/video and audio may be presented tothe user via the interface 302.

The processor 308 handles data processing for the various components ofthe head-mounted system 300 as well as data exchange between thehead-mounted system 300 and the gateway component 140 and, in someembodiments, the computing network 105. For example, the processor 308may be used to buffer and process data streaming between the user andthe computing network 105, thereby enabling a smooth, continuous andhigh fidelity user experience. In some embodiments, the processor 308may process data at a rate sufficient to achieve anywhere between 8frames/second at 320×240 resolution to 24 frames/second at highdefinition resolution (1280×720), or greater, such as 60-120frames/second and 4k resolution and higher (10k+resolution and 50,000frames/second). Additionally, the processor 308 may store and/or processdata that may be presented to the user, rather than streamed inreal-time from the computing network 105. For example, the processor 308may, in some embodiments, receive compressed data from the computingnetwork 105 and perform advanced rendering techniques (such as lightingor shading) to alleviate the data load transmitted to the user device120 from the computing network 105. In another example, the processor308 may store and/or process local object data rather than transmittingthe data to the gateway component 140 or to the computing network 105.

The head-mounted system 300 may, in some embodiments, include varioussettings, or modes, that allow for a variety of visual/interfaceperformance and functionality. The modes may be selected manually by theuser, or automatically by components of the head-mounted system 300 orthe gateway component 140. As previously mentioned, one example ofhead-mounted system 300 includes an “off” mode, wherein the interface302 provides substantially no digital or virtual content. In the offmode, the display component 303 may be transparent, thereby enabling anoptically correct view of the physical environment around the user withlittle-to-no optical distortion or computing overlay.

In one example embodiment, the head-mounted system 300 includes an“augmented” mode, wherein the interface 302 provides an augmentedreality interface. In the augmented mode, the interface display 303 maybe substantially transparent, thereby allowing the user to view thelocal, physical environment. At the same time, virtual object dataprovided by the computing network 105, the processor 308, and/or thegateway component 140 is presented on the display 303 in combinationwith the physical, local environment.

FIG. 4 illustrates an example embodiment of objects viewed by a userwhen the interface 302 is operating in an augmented mode. As shown inFIG. 4, the interface 302 presents a physical object 402 and a virtualobject 404. In the embodiment illustrated in FIG. 4, the physical object402 is a real, physical object existing in the local environment of theuser, whereas the virtual object 404 is an object created by the system100, and displayed via the user interface 302. In some embodiments, thevirtual object 404 may be displayed at a fixed position or locationwithin the physical environment (e.g., a virtual monkey standing next toa particular street sign located in the physical environment), or may bedisplayed to the user as an object located at a position relative to theuser interface/display 303 (e.g., a virtual clock or thermometer visiblein the upper, left corner of the display 303).

In some embodiments, virtual objects may be made to be cued off of, ortrigged by, an object physically present within or outside a user'sfield of view. Virtual object 404 is cued off, or triggered by, thephysical object 402. For example, the physical object 402 may actuallybe a stool, and the virtual object 404 may be displayed to the user(and, in some embodiments, to other users interfacing the system 100) asa virtual animal standing on the stool. In such an embodiment, theenvironment-sensing system 306 may use software and/or firmware stored,for example, in the processor 308 to recognize various features and/orshape patterns (captured by the sensors 312) to identify the physicalobject 402 as a stool. These recognized shape patterns such as, forexample, the stool top, may be used to trigger the placement of thevirtual object 404. Other examples include walls, tables, furniture,cars, buildings, people, floors, plants, animals—any object which can beseen can be used to trigger an augmented reality experience in somerelationship to the object or objects.

In some embodiments, the particular virtual object 404 that is triggeredmay be selected by the user or automatically selected by othercomponents of the head-mounted system 300 or interface system 100.Additionally, in embodiments in which the virtual object 404 isautomatically triggered, the particular virtual object 404 may beselected based upon the particular physical object 402 (or featurethereof) off which the virtual object 404 is cued or triggered. Forexample, if the physical object is identified as a diving boardextending over a pool, the triggered virtual object may be a creaturewearing a snorkel, bathing suit, floatation device, or other relateditems.

In another example embodiment, the head-mounted system 300 may include a“virtual” mode, wherein the interface 302 provides a virtual realityinterface. In the virtual mode, the physical environment is omitted fromthe display 303, and virtual object data provided by the computingnetwork 105, the processor 308, and/or the gateway component 140 ispresented on the display 303. The omission of the physical environmentmay be accomplished by physically blocking the visual display 303 (e.g.,via a cover) or through a feature of the interface 302 wherein thedisplay 303 transitions to an opaque setting. In the virtual mode, liveand/or stored visual and audio sensory may be presented to the userthrough the interface 302, and the user experiences and interacts with adigital world (digital objects, other users, etc.) through the virtualmode of the interface 302. Thus, the interface provided to the user inthe virtual mode is comprised of virtual object data comprising avirtual, digital world.

FIG. 5 illustrates an example embodiment of a user interface when thehead-mounted interface 302 is operating in a virtual mode. As shown inFIG. 5, the user interface presents a virtual world 500 comprised ofdigital objects 510, wherein the digital objects 510 may includeatmosphere, weather, terrain, buildings, and people. Although it is notillustrated in FIG. 5, digital objects may also include, for example,plants, vehicles, animals, creatures, machines, artificial intelligence,location information, and any other object or information defining thevirtual world 500.

In another example embodiment, the head-mounted system 300 may include a“blended” mode, wherein various features of the head-mounted system 300(as well as features of the virtual and augmented modes) may be combinedto create one or more custom interface modes. In one example custominterface mode, the physical environment is omitted from the display303, and virtual object data is presented on the display 303 in a mannersimilar to the virtual mode. However, in this example custom interfacemode, virtual objects may be fully virtual (i.e., they do not exist inthe local, physical environment) or they may be real, local, physicalobjects rendered as a virtual object in the interface 302 in place ofthe physical object. Thus, in this particular custom mode (referred toherein as a blended virtual interface mode), live and/or stored visualand audio sensory may be presented to the user through the interface302, and the user experiences and interacts with a digital worldcomprising fully virtual objects and rendered physical objects.

FIG. 6 illustrates an example embodiment of a user interface operatingin accordance with the blended virtual interface mode. As shown in FIG.6, the user interface presents a virtual world 600 comprised of fullyvirtual objects 610, and rendered physical objects 620 (renderings ofobjects otherwise physically present in the scene). In accordance withthe example illustrated in FIG. 6, the rendered physical objects 620include a building 620A, ground 620B, and a platform 620C, and are shownwith a bolded outline 630 to indicate to the user that the objects arerendered. Additionally, the fully virtual objects 610 include anadditional user 610A, clouds 610B, sun 610C, and flames 610D on top ofthe platform 620C. It should be appreciated that fully virtual objects610 may include, for example, atmosphere, weather, terrain, buildings,people, plants, vehicles, animals, creatures, machines, artificialintelligence, location information, and any other object or informationdefining the virtual world 600, and not rendered from objects existingin the local, physical environment. Conversely, the rendered physicalobjects 620 are real, local, physical objects rendered as a virtualobject in the interface 302. The bolded outline 630 represents oneexample for indicating rendered physical objects to a user. As such, therendered physical objects may be indicated as such using methods otherthan those disclosed herein.

In some embodiments, the rendered physical objects 620 may be detectedusing the sensors 312 of the environment-sensing system 306 (or usingother devices such as a motion or image capture system), and convertedinto digital object data by software and/or firmware stored, forexample, in the processing circuitry 308. Thus, as the user interfaceswith the system 100 in the blended virtual interface mode, variousphysical objects may be displayed to the user as rendered physicalobjects. This may be especially useful for allowing the user tointerface with the system 100, while still being able to safely navigatethe local, physical environment. In some embodiments, the user may beable to selectively remove or add the rendered physical objects to theinterface display 303.

In another example custom interface mode, the interface display 303 maybe substantially transparent, thereby allowing the user to view thelocal, physical environment, while various local, physical objects aredisplayed to the user as rendered physical objects. This example custominterface mode is similar to the augmented mode, except that one or moreof the virtual objects may be rendered physical objects as discussedabove with respect to the previous example.

The foregoing example custom interface modes represent a few exampleembodiments of various custom interface modes capable of being providedby the blended mode of the head-mounted system 300. Accordingly, variousother custom interface modes may be created from the various combinationof features and functionality provided by the components of thehead-mounted system 300 and the various modes discussed above withoutdeparting from the scope of the present disclosure.

The embodiments discussed herein merely describe a few examples forproviding an interface operating in an off, augmented, virtual, orblended mode, and are not intended to limit the scope or content of therespective interface modes or the functionality of the components of thehead-mounted system 300. For example, in some embodiments, the virtualobjects may include data displayed to the user (time, temperature,elevation, etc.), objects created and/or selected by the system 100,objects created and/or selected by a user, or even objects representingother users interfacing the system 100. Additionally, the virtualobjects may include an extension of physical objects (e.g., a virtualsculpture growing from a physical platform) and may be visuallyconnected to, or disconnected from, a physical object.

The virtual objects may also be dynamic and change with time, change inaccordance with various relationships (e.g., location, distance, etc.)between the user or other users, physical objects, and other virtualobjects, and/or change in accordance with other variables specified inthe software and/or firmware of the head-mounted system 300, gatewaycomponent 140, or servers 110. For example, in certain embodiments, avirtual object may respond to a user device or component thereof (e.g.,a virtual ball moves when a haptic device is placed next to it),physical or verbal user interaction (e.g., a virtual creature runs awaywhen the user approaches it, or speaks when the user speaks to it), achair is thrown at a virtual creature and the creature dodges the chair,other virtual objects (e.g., a first virtual creature reacts when itsees a second virtual creature), physical variables such as location,distance, temperature, time, etc. or other physical objects in theuser's environment (e.g., a virtual creature shown standing in aphysical street becomes flattened when a physical car passes).

The various modes discussed herein may be applied to user devices otherthan the head-mounted system 300. For example, an augmented realityinterface may be provided via a mobile phone or tablet device. In suchan embodiment, the phone or tablet may use a camera to capture thephysical environment around the user, and virtual objects may beoverlaid on the phone/tablet display screen. Additionally, the virtualmode may be provided by displaying the digital world on the displayscreen of the phone/tablet. Accordingly, these modes may be blended asto create various custom interface modes as described above using thecomponents of the phone/tablet discussed herein, as well as othercomponents connected to, or used in combination with, the user device.For example, the blended virtual interface mode may be provided by acomputer monitor, television screen, or other device lacking a cameraoperating in combination with a motion or image capture system. In thisexample embodiment, the virtual world may be viewed from themonitor/screen and the object detection and rendering may be performedby the motion or image capture system.

FIG. 7 illustrates an example embodiment of the present disclosure,wherein two users located in different geographical locations eachinteract with the other user and a common virtual world through theirrespective user devices. In this embodiment, the two users 701 and 702are throwing a virtual ball 703 (a type of virtual object) back andforth, wherein each user is capable of observing the impact of the otheruser on the virtual world (e.g., each user observes the virtual ballchanging directions, being caught by the other user, etc.). Since themovement and location of the virtual objects (i.e., the virtual ball703) are tracked by the servers 110 in the computing network 105, thesystem 100 may, in some embodiments, communicate to the users 701 and702 the exact location and timing of the arrival of the ball 703 withrespect to each user. For example, if the first user 701 is located inLondon, the user 701 may throw the ball 703 to the second user 702located in Los Angeles at a velocity calculated by the system 100.Accordingly, the system 100 may communicate to the second user 702(e.g., via email, text message, instant message, etc.) the exact timeand location of the ball's arrival. As such, the second user 702 may usehis device to see the ball 703 arrive at the specified time and located.One or more users may also use geo-location mapping software (orsimilar) to track one or more virtual objects as they travel virtuallyacross the globe. An example of this may be a user wearing a 3Dhead-mounted display looking up in the sky and seeing a virtual planeflying overhead, superimposed on the real world. The virtual plane maybe flown by the user, by intelligent software agents (software runningon the user device or gateway), other users who may be local and/orremote, and/or any of these combinations.

As previously mentioned, the user device may include a haptic interfacedevice, wherein the haptic interface device provides a feedback (e.g.,resistance, vibration, lights, sound, etc.) to the user when the hapticdevice is determined by the system 100 to be located at a physical,spatial location relative to a virtual object. For example, theembodiment described above with respect to FIG. 7 may be expanded toinclude the use of a haptic device 802, as shown in FIG. 8. In thisexample embodiment, the haptic device 802 may be displayed in thevirtual world as a baseball bat. When the ball 703 arrives, the user 702may swing the haptic device 802 at the virtual ball 703. If the system100 determines that the virtual bat provided by the haptic device 802made “contact” with the ball 703, then the haptic device 802 may vibrateor provide other feedback to the user 702, and the virtual ball 703 mayricochet off the virtual bat in a direction calculated by the system 100in accordance with the detected speed, direction, and timing of theball-to-bat contact.

The disclosed system 100 may, in some embodiments, facilitate mixed modeinterfacing, wherein multiple users may interface a common virtual world(and virtual objects contained therein) using different interface modes(e.g., augmented, virtual, blended, etc.). For example, a first userinterfacing a particular virtual world in a virtual interface mode mayinteract with a second user interfacing the same virtual world in anaugmented reality mode.

FIG. 9A illustrates an example wherein a first user 901 (interfacing adigital world of the system 100 in a blended virtual interface mode) andfirst object 902 appear as virtual objects to a second user 922interfacing the same digital world of the system 100 in a full virtualreality mode. As described above, when interfacing the digital world viathe blended virtual interface mode, local, physical objects (e.g., firstuser 901 and first object 902) may be scanned and rendered as virtualobjects in the virtual world. The first user 901 may be scanned, forexample, by a motion capture system or similar device, and rendered inthe virtual world (by software/firmware stored in the motion capturesystem, the gateway component 140, the user device 120, system servers110, or other devices) as a first rendered physical object 931.Similarly, the first object 902 may be scanned, for example, by theenvironment-sensing system 306 of a head-mounted interface 300, andrendered in the virtual world (by software/firmware stored in theprocessor 308, the gateway component 140, system servers 110, or otherdevices) as a second rendered physical object 932. The first user 901and first object 902 are shown in a first portion 910 of FIG. 9A asphysical objects in the physical world. In a second portion 920 of FIG.9A, the first user 901 and first object 902 are shown as they appear tothe second user 922 interfacing the same digital world of the system 100in a full virtual reality mode: as the first rendered physical object931 and second rendered physical object 932.

FIG. 9B illustrates another example embodiment of mixed modeinterfacing, wherein the first user 901 is interfacing the digital worldin a blended virtual interface mode, as discussed above, and the seconduser 922 is interfacing the same digital world (and the second user'sphysical, local environment 925) in an augmented reality mode. In theembodiment in FIG. 9B, the first user 901 and first object 902 arelocated at a first physical location 915, and the second user 922 islocated at a different, second physical location 925 separated by somedistance from the first location 915. In this embodiment, the virtualobjects 931 and 932 may be transposed in real-time (or near real-time)to a location within the virtual world corresponding to the secondlocation 925. Thus, the second user 922 may observe and interact, in thesecond user's physical, local environment 925, with the renderedphysical objects 931 and 932 representing the first user 901 and firstobject 902, respectively.

FIG. 10 illustrates an example illustration of a user's view wheninterfacing the system 100 in an augmented reality mode. As shown inFIG. 10, the user sees the local, physical environment (i.e., a cityhaving multiple buildings) as well as a virtual character 1010 (i.e.,virtual object). The position of the virtual character 1010 may betriggered by a 2D visual target (for example, a billboard, postcard ormagazine) and/or one or more 3D reference frames such as buildings,cars, people, animals, ai_(rp)lanes, portions of a building, and/or any3D physical object, virtual object, and/or combinations thereof. In theexample illustrated in FIG. 10, the known position of the buildings inthe city may provide the registration fiducials and/or information andkey features for rendering the virtual character 1010. Additionally, theuser's geospatial location (e.g., provided by GPS, attitude/positionsensors, etc.) or mobile location relative to the buildings, maycomprise data used by the computing network 105 to trigger thetransmission of data used to display the virtual character(s) 1010. Insome embodiments, the data used to display the virtual character 1010may comprise the rendered character 1010 and/or instructions (to becarried out by the gateway component 140 and/or user device 120) forrendering the virtual character 1010 or portions thereof. In someembodiments, if the geospatial location of the user is unavailable orunknown, a server 110, gateway component 140, and/or user device 120 maystill display the virtual object 1010 using an estimation algorithm thatestimates where particular virtual objects and/or physical objects maybe located, using the user's last known position as a function of timeand/or other parameters. This may also be used to determine the positionof any virtual objects should the user's sensors become occluded and/orexperience other malfunctions.

In some embodiments, virtual characters or virtual objects may comprisea virtual statue, wherein the rendering of the virtual statue istriggered by a physical object. For example, referring now to FIG. 11, avirtual statue 1110 may be triggered by a real, physical platform 1120.The triggering of the statue 1110 may be in response to a visual objector feature (e.g., fiducials, design features, geometry, patterns,physical location, altitude, etc.) detected by the user device or othercomponents of the system 100. When the user views the platform 1120without the user device, the user sees the platform 1120 with no statue1110. However, when the user views the platform 1120 through the userdevice, the user sees the statue 1110 on the platform 1120 as shown inFIG. 11. The statue 1110 is a virtual object and, therefore, may bestationary, animated, change over time or with respect to the user'sviewing position, or even change depending upon which particular user isviewing the statue 1110. For example, if the user is a small child, thestatue may be a dog; yet, if the viewer is an adult male, the statue maybe a large robot as shown in FIG. 11. These are examples of userdependent and/or state dependent experiences. This will enable one ormore users to perceive one or more virtual objects alone and/or incombination with physical objects and experience customized andpersonalized versions of the virtual objects. The statue 1110 (orportions thereof) may be rendered by various components of the systemincluding, for example, software/firmware installed on the user device.Using data indicating the location and attitude of the user device, incombination with the registration features of the virtual object (i.e.,statue 1110), the virtual object (i.e., statue 1110) forms arelationship with the physical object (i.e., platform 1120). Forexample, the relationship between one or more virtual objects with oneor more physical objects may be a function of distance, positioning,time, geo-location, proximity to one or more other virtual objects,and/or any other functional relationship that includes virtual and/orphysical data of any kind. In some embodiments, image recognitionsoftware in the user device may further enhance the digital-to-physicalobject relationship.

The interactive interface provided by the disclosed system and methodmay be implemented to facilitate various activities such as, forexample, interacting with one or more virtual environments and objects,interacting with other users, as well as experiencing various forms ofmedia content, including advertisements, music concerts, and movies.Accordingly, the disclosed system facilitates user interaction such thatthe user not only views or listens to the media content, but rather,actively participates in and experiences the media content. In someembodiments, the user participation may include altering existingcontent or creating new content to be rendered in one or more virtualworlds. In some embodiments, the media content, and/or users creatingthe content, may be themed around a mythopoeia of one or more virtualworlds.

In one example, musicians (or other users) may create musical content tobe rendered to users interacting with a particular virtual world. Themusical content may include, for example, various singles, EPs, albums,videos, short films, and concert performances. In one example, a largenumber of users may interface the system 100 to simultaneouslyexperience a virtual concert performed by the musicians.

In some embodiments, the media produced may contain a unique identifiercode associated with a particular entity (e.g., a band, artist, user,etc.). The code may be in the form of a set of alphanumeric characters,UPC codes, QR codes, 2D image triggers, 3D physical object featuretriggers, or other digital mark, as well as a sound, image, and/or both.In some embodiments, the code may also be embedded with digital mediawhich may be interfaced using the system 100. A user may obtain the code(e.g., via payment of a fee) and redeem the code to access the mediacontent produced by the entity associated with the identifier code. Themedia content may be added or removed from the user's interface.

The embodiments disclosed herein are provided to illustrate one or moreexamples of methods and apparatus for enabling interactive virtual oraugmented reality environments for multiple users. As such, variationsto the methods and apparatus disclosed herein may be made withoutdeparting from the scope of the present disclosure as set forth in theclaims provided below. For example, although various examples andembodiments are discussed herein with respect to a head-mounted displaysystem, the various examples and embodiments may also apply to otheruser devices capable of providing the interface or capabilitiesdiscussed with respect to those particular embodiments.

What is claimed is:
 1. A system for presenting virtual content, the system comprising: a wearable user device configured to display virtual content, a display operatively coupled to the wearable user device and configured to present the virtual content to a user, memory, processing circuitry, software stored in the memory and executable by the processing circuitry to render the virtual content for presentation by the display, a gateway operatively coupled to, and distinct from, the wearable user device and configured to monitor and regulate data representing the rendered virtual content to be displayed by the software; and a user sensing system connected to the wearable user device comprising a camera configured to detect an angular measurement of at least one eye pupil of the user, wherein the gateway monitors and regulates the display of the data representing the rendered virtual content to allow an optimum data processing of the wearable user device based on the detected head pose or angular measurement, wherein the system is configured such that at least a portion of the virtual world changes in response to a change in the virtual world data, wherein, in conjunction with the virtual world changes in response to a change in the virtual world data, at least a portion of the virtual world data is changed in response to a static physical object external to the user and sensed by the wearable user device, wherein the static physical object external to the user comprises a mapped object in a physical environment in vicinity of the first user, wherein the optimum data processing comprises prioritizing a plurality of renderings, such that processing a rendering of a dynamic virtual object is prioritized over a rendering of a static virtual object and processing a rendering of data in a field of view having less than sixty degrees forward of the angular measurement of the user's pupil is prioritized over a rendering of data outside the field of view, where the plurality of renderings are performed by the gateway device and transmitted to the wearable user device, and wherein the change in virtual world data represents rendering at least one of the dynamic virtual object and the static virtual object with the static physical object external to the user according to a predetermined relationship.
 2. The system of claim 1, wherein the optimum data processing is determined by a type of the virtual content data.
 3. The system of claim 2, wherein the type is a dynamic virtual object and the gateway transmits the virtual content over a high priority data stream.
 4. The system of claim 2, wherein the type is a static object and the gateway stores the virtual content in a local memory buffer.
 5. The system of claim 1, wherein the optimum data processing is determined by a location of the virtual content within a user's field of view.
 6. The system of claim 5, wherein the location is a central portion of the user's field of view and the gateway transmits virtual world data over a high priority data stream.
 7. The system of claim 5, wherein the location is a periphery of the user's field of view and the gateway transmits virtual world data to a local memory buffer.
 8. The system according to claim 1, wherein the change in virtual world data is presented to a second user device for presentation to a second user according to the predetermined relationship.
 9. The system according to claim 1, wherein the virtual world is operable to be rendered by at least one of the computer servers or the wearable user device.
 10. The system according to claim 1, wherein the virtual world is presented in at least one of a two-dimensional format or three-dimensional format.
 11. The system according to claim 1, wherein the wearable user device is operable to provide an interface for enabling interaction between a user and the virtual world in at least one of an augmented reality mode, a virtual reality mode, or a combination of augmented and virtual reality mode.
 12. The system according to claim 1, wherein the virtual world data is transmitted over a data network.
 13. The system according to claim 1, wherein the computer network is operable to receive at least a portion of the virtual world data from the wearable user device.
 14. The system according to claim 1, wherein at least a portion of the virtual world data transmitted to the wearable user device comprises instructions for generating at least a portion of the virtual world.
 15. The system of claim 1, wherein the wearable device is configured to be a head-worn device.
 16. The system of claim 1, further comprising an environment-sensing system coupled to the wearable user device configured to obtain data from the physical environment around a user.
 17. The system of claim 16, wherein the environment-sensing system detects key features of the static physical object external to the user.
 18. The system of claim 16, wherein the computer network receives data from the environment-sensing system to trigger a transmission of the virtual world data to the wearable user device. 